Automatic Emotion Recognition Research Articles

Emotion and Gesture Recognition

Human Gestures and emotions play an important role in interpersonal relationships. The automatic recognition of emotions and gestures has been an active research topic from early eras. Emotions are reflected from facial expression, speech and gesture of the body. Hence understanding of emotion and gesture has high importance in interaction between humans(human-human) as well as between human and machine communication.In this system captured images are compared with the trained dataset available in the database and then display emotional state and gesture. For this system, fer13 (Facial Expressions Recognition 2013 dataset) and VGG16 (Visual Geometry Group) datasets are used to compare with captured images. In this system emotion and gesture recognition, both will run simultaneously. EDA(Exploratory Data Analysis) is used to analyse and validate the train data for both emotion as well as gesture dataset.

Emotion recognition with reduced channels using CWT based EEG feature representation and a CNN classifier

Objective. Although emotion recognition has been studied for decades, a more accurate classification method that requires less computing is still needed. At present, in many studies, EEG features are extracted from all channels to recognize emotional states, however, there is a lack of an efficient feature domain that improves classification performance and reduces the number of EEG channels. Approach. In this study, a continuous wavelet transform (CWT)-based feature representation of multi-channel EEG data is proposed for automatic emotion recognition. In the proposed feature, the time-frequency domain information is preserved by using CWT coefficients. For a particular EEG channel, each CWT coefficient is mapped into a strength-to-entropy component ratio to obtain a 2D representation. Finally, a 2D feature matrix, namely CEF2D, is created by concatenating these representations from different channels and fed into a deep convolutional neural network architecture. Based on the CWT domain energy-to-entropy ratio, effective channel and CWT scale selection schemes are also proposed to reduce computational complexity. Main results. Compared with previous studies, the results of this study show that valence and arousal classification accuracy has improved in both 3-class and 2-class cases. For the 2-class problem, the average accuracies obtained for valence and arousal dimensions are 98.83% and 98.95%, respectively, and for the 3-class, the accuracies are 98.25% and 98.68%, respectively. Significance. Our findings show that the entropy-based feature of EEG data in the CWT domain is effective for emotion recognition. Utilizing the proposed feature domain, an effective channel selection method can reduce computational complexity.

Machine Learning Based Speech Emotion Recognition System

In the last decade, there has been significant research into Automatic Speech Emotion Recognition (SER). The primary goal of SER is to improve human-machine interfaces. It can also monitor someone's psychological state for lie detection applications. Recently, speech emotion recognition has found uses in medicine and forensics. This paper recognizes 7 emotions using pitch and prosody features. The majority of speech features used here are in the time domain. A Support Vector Machine (SVM) classifier categorizes the emotions. The Berlin emotional database was used for this task. A good recognition rate of 81% was achieved. The reference paper for this work recognized 4 emotions and obtained a 94.2% recognition rate. However, the reference paper used a more complex hybrid classifier, while this work focuses on recognizing more emotions with a simpler model. . KeyWords: Blindness, Smart Stick, BlinDar, GPS, ESP8266, Internet of Things, RF Tx/Rx, MQ2.

HERF: A Machine Learning Framework for Automatic Emotion Recognition from Audio

HERF: A Machine Learning Framework for Automatic Emotion Recognition from Audio

A multi-featured expression recognition model incorporating attention mechanism and object detection structure for psychological problem diagnosis

Expression is the main method for judging the emotional state and psychological condition of the human body, and the prediction of changes in facial expressions can effectively determine the mental health of a person, thus avoiding serious psychological or psychiatric disorders due to early negligence. From a computer vision perspective, most researchers have focused on studying facial expression analysis, and in some cases, body posture is also considered. However their performance is more limited under unconstrained natural conditions, which requires more information to be used in human emotion analysis. In this paper, we design an Adaptive Multi-End Fusion Attention Mechanism suitable for extracting human body information based on the deep learning framework, depending on human expressions, postures and the environment they are in and add it to an object detection model to obtain the information we need from different regions of the human body and face and features of different sizes and use fusion networks for feature fusion and classification, and from different test methods to confirm that this fusion approach for expression recognition and prediction is feasible. This model achieves an average accuracy of 34.51 % in the Emotic contextual expression recognition dataset.

Evaluation of Eye-Blinking Dynamics in Human Emotion Recognition Using Weighted Visibility Graph

Purpose: Designing an automated emotion recognition system using biosignals has become a hot and challenging issue in many fields, including human-computer interferences, robotics, and affective computing. Several algorithms have been proposed to characterize the internal and external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are habitually analyzed in a multi-modality system using classic statistical measures, and the evaluation of its dynamics has been neglected so far.
 Materials and Methods: This experiment intended to provide an innovative single-modality scheme for emotion classification using eye-blinking data. The dynamics of eye-blinking data have been characterized by weighted visibility graph-based indices. The extracted measures were then fed to the different classifiers, including support vector machine, decision tree, k-Nearest neighbor, Adaptive Boosting, and random subset to complete the process of classifying sad, happy, neutral, and fearful affective states. The scheme has been evaluated utilizing the available signals in the SEED-IV database.
 Results: The proposed framework provided significant performance in terms of recognition rates. The highest average recognition rates of  > 90% were achieved using the decision tree.
 Conclusion: In brief, our results showed that eye-blinking data has the potential for emotion recognition. The present system can be extended for designing future affect recognition systems.

Emotional Recognition Based on Faces through Deep Learning Algorithms

Facial expressions have long been a straightforward way for humans to determine emotions, but computer systems find it significantly more difficult to do the same. Emotion recognition from facial expressions, a subfield of social signal processing, is employed in many different circumstances, but is especially useful for human-computer interaction. Many studies have been conducted on automatic emotion recognition, with the majority utilizing machine learning techniques. However, the identification of basic emotions such as fear, sadness, surprise, anger, happiness, and contempt remains a challenging subject in computer vision. Recently, deep learning has gained more attention as potential solutions for a range of real-world problems, such as emotion recognition. In this work, we refined the convolutional neural network method to discern seven basic emotions and assessed several preprocessing approaches to illustrate their impact on CNN performance. The goal of this research is to enhance facial emotions and features by using emotional recognition. Computers may be able to forecast mental states more accurately and respond with more customised answers if they can identify or recognise the facial expressions that elicit human responses. Consequently, we investigate how a convolutional neural network-based deep learning technique may enhance the recognition of emotions from facial features (CNN). Consequently, we investigate how a convolutional neural network-based deep learning technique may enhance the recognition of emotions from facial features (CNN). Our dataset, which comprises of roughly 32,298 pictures for testing and training, includes multiple face expressions. After noise removal from the input image, the pretraining phase helps reveal face detection, including feature extraction. The preprocessing system helps with this.

Deep Learning, Ensemble and Supervised Machine Learning for Arabic Speech Emotion Recognition

Today, automatic emotion recognition in speech is one of the most important areas of research in signal processing. Identifying emotional content in Arabic speech is regarded as a very challenging and intricate task due to several obstacles, such as the wide range of cultures and dialects, the influence of cultural factors on emotional expression, and the scarcity of available datasets. This study used a variety of artificial intelligence models, including Xgboost, Adaboost, KNN, DT, and SOM, and a deep-learning model named SERDNN. ANAD was employed as a training dataset, which contains three emotions, "angry", "happy", and "surprised", with 844 features. This study aimed to present a more efficient and accurate technique for recognizing emotions in Arabic speech. Precision, accuracy, recall, and F1-score metrics were utilized to evaluate the effectiveness of the proposed techniques. The results showed that the Xgboost, SOM, and KNN classifiers achieved superior performance in recognizing emotions in Arabic speech. The SERDNN deep learning model outperformed the other techniques, achieving the highest accuracy of 97.40% with a loss rate of 0.1457. Therefore, it can be relied upon and deployed to recognize emotions in Arabic speech.

Enhanced multimodal emotion recognition in healthcare analytics: A deep learning based model-level fusion approach

Deep learning techniques have drawn considerable interest in emotion recognition due to recent technological developments in healthcare analytics. Automatic patient emotion recognition can assist healthcare analytics by providing feedback to the stakeholders of competent healthcare about the conditions of the patients and their satisfaction levels. In this paper, we propose a novel model-level fusion technique based on deep learning for enhanced emotion recognition from multimodal signals to monitor patients in connected healthcare. The representative visual features from the video signals are extracted through the Depthwise Separable Convolution Neural Network, and the optimized temporal attributes are derived from the multiple physiological data utilizing Bi-directional Long Short-Term Memory. A soft attention method fused the high multimodal features obtained from the two data modalities to retrieve the most significant features by focusing on emotionally salient parts of the features. We exploited two face detection methods, Histogram of Oriented Gradients and Convolutional Neural Network-based face detector (ResNet-34), to observe the effects of facial features on emotion recognition. Lastly, extensive experimental evaluations have been conducted using the widely used Bio Vid Emo DB multimodal dataset to verify the performance of the proposed architecture. Experimental results show that the developed fusion architecture improved the accuracy of emotion recognition from multimodal signals and outperformed the performance of both state-of-the-art techniques and baseline methods.

Automatic classification of emotions in speech: methods and data

The subject of this study is the data and methods used in the task of automatic recognition of emotions in spoken speech. This task has gained great popularity recently, primarily due to the emergence of large datasets of labeled data and the development of machine learning models. The classification of speech utterances is usually based on 6 archetypal emotions: anger, fear, surprise, joy, disgust and sadness. Most modern classification methods are based on machine learning and transformer models using a self-learning approach, in particular, models such as Wav2vec 2.0, HuBERT and WavLM, which are considered in this paper. English and Russian datasets of emotional speech, in particular, the datasets Dusha and RESD, are analyzed as data. As a method, an experiment was conducted in the form of comparing the results of Wav2vec 2.0, HuBERT and WavLM models applied to the relatively recently collected Russian datasets of emotional speech Dusha and RESD. The main purpose of the work is to analyze the availability and applicability of available data and approaches to recognizing emotions in speech for the Russian language, for which relatively little research has been conducted up to this point. The best result was demonstrated by the WavLM model on the Dusha dataset - 0.8782 dataset according to the Accuracy metric. The WavLM model also received the best result on the RESD dataset, while preliminary training was conducted for it on the Dusha - 0.81 dataset using the Accuracy metric. High classification results, primarily due to the quality and size of the collected Dusha dataset, indicate the prospects for further development of this area for the Russian language.

Composing music based on machine learning

As a matter of fact, the music is an important tool and pathway for human emotional communication, with great significance, especially in recent years. However, it should be noted that traditional music production methods require a lot of time, manpower, and funds, and have high costs. With the rapid development of machine learning technology as well as computation ability, the computer automatic composition based on machine learning technology is receiving more and more attention. With this in mind, this study introduces and discusses the automatic emotion recognition and emotion based music generation based on this advanced technology, as well as describes the process involved. According to the analysis, the advanced applications and implementations results have been demonstrated. At the same time, the current limitations as well as the future prospects will be clarified and discussed. Overall, these results shed light on guiding further exploration of machine learning implementations of music composition.

A Survey on Speech Emotion Recognition System using CNN Algorithm

The speech emotion recognition system (SER) plays an important role in decoding and predicting the speaker's emotional state by analyzing audio signals. Emotions are often simplified and grouped into categories such as anger, happiness, sadness, and even neutral emotional states. These emotions serve to communicate one's thoughts and provide insight into one's physical and mental health. Speech is the most basic and natural form of communication when interacting with others. Voice signals enable rapid communication between individuals, making them a valuable and effective method of expression. Over the past decades, countless research initiatives have been devoted to the development of voice-based automatic emotion recognition systems, especially to improve human-machine communication. Voice is gradually taking center stage in the field of Human-Machine interfaces in the IT sector. This interdisciplinary field draws on a variety of fields, including computer science, signal processing, psychology, linguistics, and more. As technology advances, it allows for seamless communication between humans and machines. Speech recognition not only interprets speech but also captures nuances in a person's tone and expressions, similar to body language. Therefore, it becomes an essential element of the Human-Machine communication system.

Emotion Classification Based on Transformer and CNN for EEG Spatial-Temporal Feature Learning.

The temporal and spatial information of electroencephalogram (EEG) signals is crucial for recognizing features in emotion classification models, but it excessively relies on manual feature extraction. The transformer model has the capability of performing automatic feature extraction; however, its potential has not been fully explored in the classification of emotion-related EEG signals. To address these challenges, the present study proposes a novel model based on transformer and convolutional neural networks (TCNN) for EEG spatial-temporal (EEG ST) feature learning to automatic emotion classification. The proposed EEG ST-TCNN model utilizes position encoding (PE) and multi-head attention to perceive channel positions and timing information in EEG signals. Two parallel transformer encoders in the model are used to extract spatial and temporal features from emotion-related EEG signals, and a CNN is used to aggregate the EEG's spatial and temporal features, which are subsequently classified using Softmax. The proposed EEG ST-TCNN model achieved an accuracy of 96.67% on the SEED dataset and accuracies of 95.73%, 96.95%, and 96.34% for the arousal-valence, arousal, and valence dimensions, respectively, for the DEAP dataset. The results demonstrate the effectiveness of the proposed ST-TCNN model, with superior performance in emotion classification compared to recent relevant studies. The proposed EEG ST-TCNN model has the potential to be used for EEG-based automatic emotion recognition.

AN OPTIMIZED RESIDUAL NETWORK FOR EMOTION RECOGNITION BASED ON A MULTI-FEATURES FUSION TECHNOLOGY AND ELECTROENCEPHALOGRAPHY SIGNALS

Presently, a significant number of students enrolled in colleges and universities encounter mental health challenges, including academic stress and difficulties in interpersonal relationships. These factors contribute to a decline in mental well-being and necessitate prompt intervention and assistance. The automation of mental health identification in students greatly benefits from the implementation of intelligent emotion recognition systems. Through the analysis of students’ emotional states, educational institutions can gain a deeper understanding of students’ mental well-being, enabling them to promptly identify and address any issues and offer more complete care and support to students. This paper proposes a method for automated emotion recognition that is driven by data. The method is developed based on the provided background information. The proposed approach incorporates a feature fusion mechanism and an attention mechanism into the ResNet-34 model. This integration enhances the model’s capability to analyze intricate details and subsequently improves its classification performance when applied to electroencephalography (EEG) signals. This paper introduces several key innovations. First, an optimization technique is applied to the input component of the model, enabling multifeature fusion. Second, an attention mechanism is incorporated after the residual network module, enabling the model to prioritize parts that contribute to classification and enhance feature extraction. Finally, the network parameters are optimized using both softmax loss and center loss functions. The findings from the analysis of the sentiment EEG public data SJTU Emotion EEG Dataset (SEED) indicate that the proposed sentiment recognition approach not only enhances the classification performance of the model on the sentiment EEG data but also improves the stability of the results. This paper presents a novel approach that enables automatic and efficient recognition of students’ emotions on commonly used platforms. The findings of this study hold significant implications for mental health assessment and detection in real-life production settings, offering substantial reference value in this domain.

Decoding emotions: Exploring the validity of sentiment analysis in psychotherapy

Objective Given the importance of emotions in psychotherapy, valid measures are essential for research and practice. As emotions are expressed at different levels, multimodal measurements are needed for a nuanced assessment. Natural Language Processing (NLP) could augment the measurement of emotions. The study explores the validity of sentiment analysis in psychotherapy transcripts. Method We used a transformer-based NLP algorithm to analyze sentiments in 85 transcripts from 35 patients. Construct and criterion validity were evaluated using self- and therapist reports and process and outcome measures via correlational, multitrait-multimethod, and multilevel analyses. Results The results provide indications in support of the sentiments’ validity. For example, sentiments were significantly related to self- and therapist reports of emotions in the same session. Sentiments correlated significantly with in-session processes (e.g., coping experiences), and an increase in positive sentiments throughout therapy predicted better outcomes after treatment termination. Discussion Sentiment analysis could serve as a valid approach to assessing the emotional tone of psychotherapy sessions and may contribute to the multimodal measurement of emotions. Future research could combine sentiment analysis with automatic emotion recognition in facial expressions and vocal cues via the Nonverbal Behavior Analyzer (NOVA). Limitations (e.g., exploratory study with numerous tests) and opportunities are discussed.

3DEmo: For Portrait Emotion Recognition with New Dataset

Emotional Expression Recognition (EER) and Facial Expression Recognition (FER) are active research areas in the affective computing field, which involves studying human emotion, recognition, and sentiment analysis. The main objective of this research is to develop algorithms that can accurately interpret and estimate human emotions from portrait images. The emotions depicted in a portrait can reflect various factors such as psychological and physiological states, the artist’s emotional responses, social and environmental aspects, and the period in which the painting was created. This task is challenging because (i) the portraits are often depicted in an artistic or stylized manner rather than realistically or naturally, (ii) the texture and color features obtained from natural faces and paintings differ, affecting the success rate of emotion recognition algorithms, and (iii) it is a new research area, where practically we do not have visual arts portrait facial emotion estimation models or datasets. To address these challenges, we need a new class of tools and a database specifically tailored to analyze portrait images. This study aims to develop art portrait emotion recognition methods and create a new digital portrait dataset containing 927 images. The proposed model is based on (i) a 3-dimensional estimation of emotions learned by a deep neural network and (ii) a novel deep learning module (3DEmo) that could be easily integrated into existing FER models. To evaluate the effectiveness of the developed models, we also tested their robustness on a facial emotion recognition dataset. The extensive simulation results show that the presented approach outperforms established methods. We expect that this dataset and the developed new tools will encourage further research in recognizing emotions in portrait paintings and predicting artists’ emotions in the painting period based on their artwork.

Supervised learning for automatic emotion recognition in Parkinson’s disease through smartwatch signals

People with Parkinson’s Disease (PwPD) usually experience several neuropsychiatric signs such as anxiety, depression, and negative emotions that contribute to disability and worsening of quality of life. Notwithstanding, the assessment of these symptoms are largely underrated, subjective and difficult due to a large overlapping with other PD symptoms, like hypomimia and bradikinesia. The aim and novelty of the current work is to study and validate a method for automatic emotion recognition in PwPD during daily living through autonomic signals acquired by acceptable and low-cost consumer technology. The best shallow learning algorithm and the best minimal feature set are individuated. 11 PwPD and 8 subjects with no history of neurological injury or illness were enrolled in the study. Participants were asked to watch video clips purposely selected to arouse emotions, and annotate arousal and valence of emotions triggered by video clips, while their heart rate, skin conductance, and temperature were recorded by a smartwatch. Smartwatch data was used for features extraction, while participants’ reported arousal and valence were used as gold-standard to train machine learning algorithms for emotion classification (low/high arousal, positive/negative valence). Different feature sets and different algorithms (i.e. decision tree (DT), random forest (RF), support vector machine (SVM), and multilayer perceptron (MLP)) were evaluated to find the best solution for each group of participant. In each group of participants, it was possible to find a combination of feature set and algorithm to reach a classification accuracy greater than 90%. The Random Forest reached the best performance in both groups and for both valence and arousal. For each classification task (valence or arousal, PwPD or controls), the best model was selected and the minimal feature set was found by performing a recursive feature elimination based on the Shapley value. A lower accuracy of appraisal emerged for arousal compared to valence. Obtained results showed the feasibility of automatic emotion recognition in PwPDs through autonomic signals. Autonomic dysfunction in PwPDs may explain the lower arousal accuracy. The findings warrant confirmation from trials on larger samples and there are open issues to be deepened in future work.

Emotion Recognition From Gait Analyses: Current Research and Future Directions

Human gait refers to a daily motion that represents not only mobility but can also be used to identify the walker by either human observers or computers. Recent studies reveal that gait even conveys information about the walker’s emotion. Individuals in different emotion states may show different gait patterns. The mapping between various emotions and gait patterns provides a new source for automated emotion recognition. Compared to traditional emotion detection biometrics, such as facial expression, speech, and physiological parameters, gait is remotely observable, more difficult to imitate, and requires less cooperation from the subject. These advantages make gait a promising source for emotion detection. This article reviews current research on gait-based emotion detection, particularly on how gait parameters can be affected by different emotion states and how the emotion states can be recognized through distinct gait patterns. We focus on the detailed methods and techniques applied in the whole process of emotion recognition: data collection, preprocessing, and classification. Finally, we discuss possible future developments of efficient and effective gait-based emotion recognition using state-of-the-art techniques in intelligent computation and big data.

Healthy and Happy? An Ethical Investigation of Emotion Recognition and Regulation Technologies (ERR) within Ambient Assisted Living (AAL)

Ambient Assisted Living (AAL) refers to technologies that track daily activities of persons in need of care to enhance their autonomy and minimise their need for assistance. New technological developments show an increasing effort to integrate automated emotion recognition and regulation (ERR) into AAL systems. These technologies aim to recognise emotions via different sensors and, eventually, to regulate emotions defined as “negative” via different forms of intervention. Although these technologies are already implemented in other areas, AAL stands out by its tendency to enable an inconspicuous 24-hour surveillance in the private living space of users who rely on the technology to maintain a certain degree of independence in their daily activities. The combination of both technologies represents a new dimension of emotion recognition in a potentially vulnerable group of users. Our paper aims to provide an ethical contextualisation of the novel combination of both technologies. We discuss different concepts of emotions, namely Basic Emotion Theory (BET) and the Circumplex Model of Affect (CMA), that form the basis of ERR and provide an overview over the current technological developments in AAL. We highlight four ethical issues that specifically arise in the context of ERR in AAL systems, namely concerns regarding (1) the reductionist view of emotions, (2) solutionism as an underlying assumption of these technologies, (3) the privacy and autonomy of users and their emotions, (4) the tendency of machine learning techniques to normalise and generalise human behaviour and emotional reactions.

A multimodal teacher speech emotion recognition method in the smart classroom

As a collection of various IoT devices, smart classroom can record various forms of teaching data and provide rich data for recognizing teachers' emotions. Recognizing and analyzing teachers' emotions can promote teachers' professional development. Nowadays, most of the automatic emotion recognition methods for teachers in smart classroom are based on facial expressions. However, since teachers usually keep smiling to mobilize the classroom atmosphere, the recognition results may not reflect the real mental state of teachers. By observing teaching videos, it is found that the prosody and text in the teachers' speech can reflect the implicit emotion of the teacher. Therefore, a multimodal teacher emotion dataset (MTED) was built based on teaching videos recorded by IoT cameras and microphones in smart classroom. A neural network combining multiple prosodic features and text content for teacher speech emotion recognition is proposed. The proposed method fills the gap in teacher speech emotion recognition, our proposed method has higher accuracy. Experimental results show that ProsodyBERT achieves 78.6 % UA4 and 66.2 % UA6 on IEMOCAP and MELD, respectively, surpassing the existing methods. The proposed method reached 82.1% UA6 on MTED self-built dataset, which is 9.6 %-21.4 % higher than that of unimodal method in teacher emotion recognition. An ablation experiment is designed and implemented on MTED dataset to explore the influence of each module in ProsodyBERT on teacher speech emotion recognition task. The experimental results in the smart classroom record show that ProsodyBERT has higher accuracy and stronger robustness than unimodal methods.